Righto. So. You're a newbie magician. You want to cast magic missile, right?
What, you expect to insta-cast it right off the bat, at max power, scrub?
Absolutely not!
You gotta learn.
The first spell you cast will take longer to cast, is more prone to failure (maybe. this might be too grindy/frustrating), and is less efficient.

So, there's a curve with your spellcasting skill:

Spellcasting skill determines speed. After you reach a certain proficiency, you will no longer be impaired.